{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ch `11`: Concept `01`\n",
    "\n",
    "## Multi RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All we need is TensorFlow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, define the constants. \n",
    "\n",
    "Let's say we're dealing with 1-dimensional vectors, and a maximum sequence size of 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_dim = 1\n",
    "seq_size = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next up, define the placeholder(s). \n",
    "\n",
    "We only need one for this simple example: the input placeholder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "input_placeholder = tf.placeholder(dtype=tf.float32, shape=[None, seq_size, input_dim])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's make a helper function to create LSTM cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_cell(state_dim):\n",
    "    return tf.contrib.rnn.LSTMCell(state_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the function and extract the cell outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope(\"first_cell\") as scope:\n",
    "    cell = make_cell(state_dim=10)\n",
    "    outputs, states = tf.nn.dynamic_rnn(cell, input_placeholder, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You know what? We can just keep stacking cells on top of each other. In a new variable scope, you can pipe the output of the previous cell to the input of the new cell. Check it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope(\"second_cell\") as scope:\n",
    "    cell2 = make_cell(state_dim=10)\n",
    "    outputs2, states2 = tf.nn.dynamic_rnn(cell2, outputs, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we wanted 5 layers of RNNs? \n",
    "\n",
    "There's a useful shortcut that the TensorFlow library supplies, called `MultiRNNCell`. Here's a helper function to use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_multi_cell(state_dim, num_layers):\n",
    "    cells = [make_cell(state_dim) for _ in range(num_layers)]\n",
    "    return tf.contrib.rnn.MultiRNNCell(cells)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the helper function in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "multi_cell = make_multi_cell(state_dim=10, num_layers=5)\n",
    "outputs5, states5 = tf.nn.dynamic_rnn(multi_cell, input_placeholder, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting a session, let's prepare some simple input to the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "input_seq = [[1], [2], [3]]\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start the session, and initialize variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "init_op = tf.global_variables_initializer()\n",
    "\n",
    "sess = tf.InteractiveSession()\n",
    "sess.run(init_op)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can run the outputs to verify that the code is sound."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 0.00141981  0.0722062  -0.08216076  0.03216607  0.02329798  0.06957388\n",
      "   -0.04552787 -0.05649291  0.05059107  0.01796713]\n",
      "  [-0.00228921  0.16875982 -0.21222715  0.0772687   0.05970224  0.16551635\n",
      "   -0.10631067 -0.13780777  0.12389956  0.05248111]\n",
      "  [-0.01359726  0.24421771 -0.33965409  0.11412902  0.0964628   0.25151449\n",
      "   -0.16440172 -0.22563797  0.18972857  0.09557904]]]\n",
      "[[[-0.00224876  0.01402885  0.00929528 -0.00392457  0.00333697 -0.00213898\n",
      "   -0.0046619  -0.01061259  0.00368386  0.00040365]\n",
      "  [-0.00939647  0.04281064  0.02773804 -0.01503811  0.01025065 -0.00612708\n",
      "   -0.01655139 -0.03407493  0.01263932  0.00136939]\n",
      "  [-0.02229579  0.07774397  0.0480789  -0.03287651  0.017735   -0.01063949\n",
      "   -0.03610384 -0.06736942  0.02458673  0.00139557]]]\n",
      "[[[  1.42336748e-05  -1.58296571e-05  -1.62987853e-05   1.59381907e-05\n",
      "     1.33105495e-05  -1.38451333e-05  -2.20941274e-05   2.54621627e-05\n",
      "     2.26147549e-05  -3.30040712e-05]\n",
      "  [  8.32868463e-05  -8.99614606e-05  -1.02287340e-04   8.68237403e-05\n",
      "     8.19651614e-05  -7.38111194e-05  -1.29947555e-04   1.52955734e-04\n",
      "     1.36927833e-04  -1.89826897e-04]\n",
      "  [  2.79068598e-04  -2.77705112e-04  -3.54834105e-04   2.59170338e-04\n",
      "     2.73422222e-04  -2.13255596e-04  -4.27101302e-04   5.09521109e-04\n",
      "     4.57556103e-04  -6.11643540e-04]]]\n"
     ]
    }
   ],
   "source": [
    "outputs_val, outputs2_val, outputs5_val = sess.run([outputs, outputs2, outputs5], \n",
    "                                                   feed_dict={input_placeholder: [input_seq]})\n",
    "print(outputs_val)\n",
    "print(outputs2_val)\n",
    "print(outputs5_val)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
